Computer Vision Applications for Autonomous Aerial Vehicles

Undoubtedly, unmanned aerial vehicles (UAVs) have experienced a great leap forward over the last decade. It is not surprising anymore to see a UAV being used to accomplish a certain task, which was previously carried out by humans or a former technology. The proliferation of special vision sensors, such as depth cameras, lidar sensors and thermal cameras, and major breakthroughs in computer vision and machine learning fields accelerated the advance of UAV research and technology. However, due to certain unique challenges imposed by UAVs, such as limited payload capacity, unreliable communication link with the ground stations and data safety, UAVs are compelled to perform many tasks on their onboard embedded processing units, which makes it difficult to readily implement the most advanced algorithms on UAVs. This thesis focuses on computer vision and machine learning applications for UAVs equipped with onboard embedded platforms, and presents algorithms that utilize data from multiple modalities. The presented work covers a broad spectrum of algorithms and applications for UAVs, such as indoor UAV perception, 3D understanding with deep learning, UAV localization, and structural inspection with UAVs. Visual guidance and scene understanding without relying on pre-installed tags or markers is the desired approach for fully autonomous navigation of UAVs in conjunction with the global positioning systems (GPS), or especially when GPS information is either unavailable or unreliable. Thus, semantic and geometric understanding of the surroundings become vital to utilize vision as guidance in the autonomous navigation pipelines. In this context, first, robust altitude measurement, safe landing zone detection and doorway detection methods are presented for autonomous UAVs operating indoors. These approaches are implemented on Google Project Tango platform, which is an embedded platform equipped with various sensors including a depth camera. Next, a modified capsule network for 3D object classification is presented with weight optimization so that the network can be fit and run on memory-constrained platforms. Then, a semantic segmentation method for 3D point clouds is developed for a more general visual perception on a UAV equipped with a 3D vision sensor. Next, this thesis presents algorithms for structural health monitoring applications involving UAVs. First, a 3D point cloud-based, drift-free and lightweight localization method is presented for depth camera-equipped UAVs that perform bridge inspection, where GPS signal is unreliable. Next, a thermal leakage detection algorithm is presented for detecting thermal anomalies on building envelopes using aerial thermography from UAVs. Then, building on our thermal anomaly identification expertise gained on the previous task, a novel performance anomaly identification metric (AIM) is presented for more reliable performance evaluation of thermal anomaly identification methods

Campus as a Lab for Computer Vision-based Heat Mapping Drones: A Case Study for Multiple Building Envelope Inspection using Unmanned Aerial Systems (UAS)

Unmanned Aerial Systems (UAS – a.k.a. drones) have evolved over the past decade as both advanced military technology and off-the-shelf consumer devices. There is a gradual shift towards public use of drones, which presents opportunities for effective remote procedures that can disrupt a variety of design disciplines. In architecture praxis, UAS equipment with remote sensing gear presents an opportunity for analysis and inspection of existing building stocks, where architects, engineers, building energy auditors as well as owners can document building performance, visualize heat transfer using infrared imaging and create digital models using 3D photogrammetry. Comprehensive energy audits are essential to maximize energy savings in buildings realized from the design and implementation of deep retrofits for building envelopes, together with energy system repairs or changes. This paper presents a methodology for employing a UAS platform to conduct rapid building envelope performance diagnostics and perform aerial assessment mapping of building energy. The investigation reviews various literature that addresses this topic, followed by the identification of a standard procedures for operating a UAS for energy audit missions. The presented framework is then tested on a university campus site to showcase: 1) visually identifying areas of thermal anomalies using a UAS equipped with IR cameras; 2) detailed inspection applied to areas of high interest to quantify envelope heat-flow using computer vision techniques. The overall precision and recall rates of 76% and 74% were achieved in the experimental results, respectively. A discussion of the findings suggests refining procedure accuracy, as a step towards automated envelope inspection